Method, system and apparatus for tracking surgical imaging devices

ABSTRACT

A computing device and methods for tracking surgical imaging devices stores a preoperative image of patient tissue registered to a frame of reference of a tracking system; and receives, from a first imaging device, a first intraoperative image of a first region of the tissue, with a finer resolution than the preoperative image. The computing device receives a position of the first imaging device from the tracking system, and registers the first intraoperative image with the frame of reference. The computing device receives, from a second imaging device, a second intraoperative image of a second region of the patient tissue, with a finer resolution than the first intraoperative image. The computing device registers the second intraoperative image to the first intraoperative image; and controls a display to present the preoperative image overlaid with the first intraoperative image, and the first intraoperative image overlaid with the second intraoperative image.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This document is a continuation application, claiming the benefit of,and priority to, U.S. patent application Ser. No. 15/514,374; filed onMar. 24, 2017, and entitled “METHOD, SYSTEM AND APPARATUS FOR TRACKINGSURGICAL IMAGING DEVICES, and International Patent Application No.PCT/CA2015/000163, filed on Mar. 17, 2015, and entitled “METHOD, SYSTEMAND APPARATUS FOR TRACKING SURGICAL IMAGING DEVICES, all of which arehereby incorporated by reference herein in their entirety.

FIELD

The present disclosure generally relates to tracking surgicalinstruments. The present disclosure specifically relates to methods,systems, apparatuses, and devices for tracking surgical imaging devices.

BACKGROUND

Medical procedures, such as brain surgery, are sometimes performed withthe aid of tracking systems to indicate, e.g., on a display, wherevarious surgical instruments are located relative to the patient. Suchtracking systems can include reflective markers on the instrumentsthemselves, which are detected by one or more cameras. For example, animaging device may be equipped with markers visible to tracking cameras,and with the help of tracking information, the images of the patientcaptured by that device can be overlaid on a larger, preoperative imageof the patient on a display.

Conventional tracking systems, however, have limited resolutions.Certain imaging devices are capable of capturing high-resolution imagesof small areas of the patient (sometimes smaller than the resolution ofthe tracking system). Some surgical tools or effectors may be capable ofaffecting the surgical area down to the cellular level. The trackingsystem may not be capable of tracking such imaging devices and toolswith sufficient accuracy to correctly represent the location of thehigh-resolution images on the preoperative image. In other words, it maybecome difficult or impossible to clearly indicate on a display exactlywhich portion of the patient is being observed or affected with thehigh-resolution images.

SUMMARY

According to an aspect of the present disclosure, a method of trackingsurgical imaging devices using a computing device is provided,comprising: storing a preoperative image of patient tissue registered toa frame of reference of a tracking system connected to the computingdevice; receiving, from a first imaging device, a first intraoperativeimage of a first region of the patient tissue; the first intraoperativeimage having a finer resolution than the preoperative image; receiving aposition of the first imaging device in the frame of reference from thetracking system, and registering the first intraoperative image with theframe of reference based on the position; receiving, from a secondimaging device, a second intraoperative image of a second region of thepatient tissue; the second region being smaller than the first regionand the second intraoperative image having a finer resolution than thefirst intraoperative image; registering the second intraoperative imageto the first intraoperative image; and controlling a display connectedto the computing device to present the preoperative image overlaid withthe first intraoperative image, and the first intraoperative imageoverlaid with the second intraoperative image.

According to another aspect of the present disclosure, a computingdevice for tracking surgical imaging devices is provided. The computingdevice comprises: a display; a memory storing a preoperative image ofpatient tissue registered to a frame of reference of a tracking systemconnected to the computing device; a processor interconnected with thememory and the display, the processor configured to: receive, from afirst imaging device, a first intraoperative image of a first region ofthe patient tissue; the first intraoperative image having a finerresolution than the preoperative image; receive a position of the firstimaging device in the frame of reference from the tracking system, andregister the first intraoperative image with the frame of referencebased on the position; receive, from a second imaging device, a secondintraoperative image of a second region of the patient tissue; thesecond region being smaller than the first region and the secondintraoperative image having a finer resolution than the firstintraoperative image; register the second intraoperative image to thefirst intraoperative image; and control the display to present thepreoperative image overlaid with the first intraoperative image, and thefirst intraoperative image overlaid with the second intraoperativeimage.

BRIEF DESCRIPTION OF THE DRAWING

Embodiments are non-limiting and described with reference to thefollowing several figures of the Drawing, in which:

FIG. 1 is a diagram illustrating an operating theatre;

FIG. 2 is a diagram illustrating a computing device of the operatingtheatre, as shown in FIG. 1;

FIG. 3 is a diagram illustrating a method of tracking surgical imagingdevices;

FIG. 4 is a diagram illustrating a preoperative image received in themethod, as shown in FIG. 3;

FIG. 5A is a diagram illustrating a first intraoperative image receivedin the method, as shown in FIG. 3;

FIG. 5B is a diagram illustrating a registration of the firstintraoperative image to the preoperative image, as shown in FIG. 4;

FIG. 6A is a diagram illustrating a second intraoperative image receivedin the method of FIG. 3;

FIG. 6B is a diagram illustrating a registration of the secondintraoperative image to the first intraoperative image, as shown in FIG.5A;

FIG. 7A is a diagram illustrating a third intraoperative image receivedin the method, as shown in FIG. 3;

FIG. 7B is a diagram illustrating a the registration of the thirdintraoperative image to the second intraoperative image, as shown inFIG. 7A;

FIG. 8 is a diagram illustrating performance of the step of displayingimages overlaid in sequence of capture in the method, as shown in FIG.3;

FIG. 9 is a diagram illustrating imaging modalities and their levels ofprecision;

FIG. 10 is a diagram illustrating relationships between visualizationdevices, positioning devices and effector devices; and

DETAILED DESCRIPTION

Various embodiments and aspects of the disclosure will be described withreference to details discussed below. The following description anddrawings are illustrative of the disclosure and are not to be construedas limiting the disclosure. Numerous specific details are described toprovide a thorough understanding of various embodiments of the presentdisclosure. However, in certain instances, well-known or conventionaldetails are not described in order to provide a concise discussion ofembodiments of the present disclosure.

As used herein, the terms, “comprises” and “comprising” are to beconstrued as being inclusive and open ended, and not exclusive.Specifically, when used in the specification and claims, the terms,“comprises” and “comprising” and variations thereof mean the specifiedfeatures, steps or components are included. These terms are not to beinterpreted to exclude the presence of other features, steps orcomponents.

Unless defined otherwise, all technical and scientific terms used hereinare intended to have the same meaning as commonly understood to one ofordinary skill in the art. Unless otherwise indicated, as used herein,the following terms are intended to have the following meanings.

As used herein the term “intraoperative” refers to an action, process,method, event or step that occurs or is carried out during at least aportion of a medical procedure. The term “preoperative” as used hereinrefers to an action, process, method, event or step that occurs or iscarried out before the medical procedure begins. The termsintraoperative and preoperative, as defined herein, are not limited tosurgical procedures, and may refer to other types of medical procedures,such as diagnostic and therapeutic procedures.

Referring to FIG. 1, this diagram illustrates a surgical operatingtheatre 100 in which a healthcare worker 102, e.g., a surgeon, operateson a patient 104, in accordance with an embodiment of the presentdisclosure. Specifically, the surgeon 102 conducts a minimally invasivesurgical procedure on the brain of patient 104. Minimally invasive brainsurgery involves the insertion and manipulation of instruments into thebrain through an opening that is significantly smaller than the portionsof skull removed to expose the brain in traditional brain surgerytechniques. The description below makes reference to the brain ofpatient 104 as an example of tissue to which the techniques herein maybe applied. Those techniques may also be applied to a wide variety ofother tissues. Thus, when the brain of the patient 104 is mentionedbelow, it is simply an example of the various tissues in connection withwhich the systems and methods herein may be implemented.

Still referring to FIG. 1, the opening through which the surgeon 102inserts and manipulates instruments is provided by an access port 106.Access port 106 typically includes a hollow cylindrical device with openends. During insertion of access port 106 into the brain (after asuitable opening has been drilled in the skull), an introducer (notshown) is generally inserted into access port 106. The introducer istypically a cylindrical device that slidably engages the internalsurface of access port 106 and bears a conical atraumatic tip to allowfor insertion of access port 106 into the sulcal folds of the brain.Following insertion of access port 106, the introducer may be removed,and access port 106 may then enable insertion and bimanual manipulationof surgical tools into the brain. Examples of such tools includesuctioning devices, scissors, scalpels, cutting devices, imagingdevices, e.g., ultrasound sensors, and the like. Additional instrumentsmay be employed to conduct the procedure that do not extend into accessport 106, such as laser ablation devices (which can emit laser lightinto access port 106).

Still referring to FIG. 1, an equipment tower 108 supports a computingdevice (not shown) such as a desktop computer, as well as one or moredisplays 110 connected to the computing device for displaying imagesprovided by the computing device. The equipment tower 108 also supportsa tracking system 112. Tracking system 112 is generally configured totrack the positions of one or more reflective markers (not shown)mounted on access port 102, any of the above-mentioned surgical toolsand instruments, or any combination thereof. Such markers, also referredto as fiducial markers, may also be mounted on patient 104, for example,at various points on the head of the patient 104. Tracking system 112may therefore include a camera, e.g., a stereo camera, and a computingdevice (either the same computing device as mentioned above or aseparate computing device) configured to locate the fiducial markers inthe images captured by the camera, and determine the spatial positionsof those markers within the operating theatre. The spatial positions maybe provided by tracking system 112 to the computing device in equipmenttower 108 for subsequent use. The positions determined by trackingsystem 112 may be provided in a frame of reference 113 (that is, acoordinate system) centered at a point of origin within the operatingroom.

Still referring to FIG. 1, the nature of the markers and the camera arenot particularly limited. For example, the camera may be sensitive toinfrared (IR) or near-infrared (NIR) light, and tracking system 112 mayinclude one or more IR emitters, e.g., IR light emitting diodes (LEDs),to shine IR light on the markers. In other examples, marker recognitionin tracking system 112 may be based on radio frequency (RF) radiation,visible light emitted from devices such as pulsed or un-pulsed LEDs,electromagnetic radiation other than IR or visible light, and the like.For RF and EM-based tracking, each object can be fitted with markershaving signatures unique to that object, and tracking system 112 caninclude antennae rather than the above-mentioned camera. Combinations ofthe above may also be employed.

Still referring to FIG. 1, each tracked object generally includes threeor more markers fixed at predefined locations on the object. Thepredefined locations, as well as the geometry of each tracked object,are configured within tracking system 112; and, thus, the trackingsystem 112 is configured to image the operating theatre, compare thepositions of any visible markers to the pre-configured geometry andmarker locations, and based on the comparison, determine which trackedobjects are present in the field of view of the camera, as well as whatpositions those objects are currently in. An example of tracking system112 is the “Polaris” system available from Northern Digital Inc.

Still referring to FIG. 1, an automated articulated arm 114, alsoreferred to as a robotic arm, carries an external scope 116, e.g.,external to patient 104. External scope 116 may be positioned overaccess port 102 by robotic arm 114, and may capture images of the brainof patient 104 for presentation on display 110. The movement of roboticarm 114 to place external scope 116 correctly over access port 102 maybe guided by tracking system 112 and the computing device in equipmenttower 108. In other words, one or both of robotic arm 114 and scope 116bear markers that are detectable by tracking system 112. As will bediscussed in greater detail below, the images from external scope 116presented on display 110 may be overlaid with other images, includingimages obtained prior to the surgical procedure. The images presented ondisplay 110 may also display virtual models of surgical instrumentspresent in the field of view of tracking system 112 (the positions andorientations of the models having been determined by tracking system 112from the positions of the markers mentioned above).

Still referring to FIG. 1, in addition to scope 116, theatre 100 caninclude one or more additional imaging devices. Such additional imagingdevices can include, for example, ultrasound probes, optical coherencetomography (OCT) probes, polarization sensitive OCT (PS-OCT) probes,micro-photo-acoustic imaging probes, spectroscopy probes, e.g., Raman orother optical spectroscopy probes, mass spectroscopy probes and thelike, and the like. In the present embodiments, such imaging devices donot bear markers that are detectable by tracking system 112. Theseadditional imaging devices can bear markers in other embodiments,however certain activities performed by the computing device inequipment tower 108 can reduce or eliminate the need for such markers onany imaging device other than scope 116.

Still referring to FIG. 1, the additional imaging devices, as well asthe tools mentioned earlier, e.g., cutting tools, laser emitters and thelike) can be handheld or mounted on one or more robotic arms in additionto robotic arm 114. In general, the robotic arms on which the abovecomponents are mounted are capable of movements with resolutionscorresponding to the resolutions of the tools they support. Thus, forexample, a laser emitter capable of targeting an area of tissue having aradius of three micrometers may be supported by a robotic arm capable ofmicrometer-level movements. Conversely, scope 116 may capture images ofpatient tissue areas of up to ten centimeters square, and thus roboticarm 114 supporting scope 116 may be capable of lower-resolutionmovements, e.g., millimeter-level).

Still referring to FIG. 1, before a procedure such as that shown (whichmay be, for example, a tumor resection), preoperative images may becollected of patient 104, or at least of the brain or other tissues ofpatient 104. Such preoperative images may be collected using any of avariety of imaging modalities, including Magnetic Resonance Imaging(MRI). During the medical procedure, additional images (referred to asintraoperative images) may be collected of the brain or other tissues ofpatient 104, using any of the above-mentioned additional imagingdevices. In general, the intraoperative images are acquired at greaterpixel densities, e.g., finer resolutions, than the preoperative images,and depict smaller regions of patient 104 than the preoperative images.For example, an intraoperative ultrasound image may depict a smallerarea of the brain of patient 104 than a preoperative MRI image, but at ahigher resolution than the MRI image. In other words, the ultrasoundimage has a greater pixel density than the MRI image.

Still referring to FIG. 1, as will be described in further detail below,the computing device housed in equipment tower 108 can perform variousactions to register intraoperative images captured with scope 116 andthe additional imaging devices with each other and with preoperativeimages, thus enabling the positions of the additional imaging devices tobe tracked without the need for markers visible to tracking system 112on those imaging devices.

Referring to FIG. 2, a computing device 200 is depicted, including acentral processing unit (also referred to as a microprocessor or simplya processor) 202 interconnected with a non-transitory computer readablestorage medium such as a memory 204, in accordance with an embodiment ofthe present disclosure. Before a discussion of the functionality of thecomputing device, a brief description of the components of the computingdevice will be provided. Processor 202 and memory 204 are generallycomprised of one or more integrated circuits (ICs), and can have avariety of structures, as will now occur to those skilled in the art(for example, more than one CPU can be provided). Memory 204 can be anysuitable combination of volatile, e.g., random access memory (“RAM”))and non-volatile, e.g., read only memory (“ROM”), electrically erasableprogrammable read-only memory (“EEPROM”), flash memory, magneticcomputer storage device, or optical disc) memory. In the presentexample, memory 204 includes both a volatile memory and a non-volatilememory. Other types of non-transitory computer readable storage mediumare also contemplated, such as compact discs (CD-ROM, CD-RW) and digitalvideo discs (DVD).

Still referring to FIG. 2, the computing device 200 also includes anetwork interface 206 interconnected with processor 202. Networkinterface 206 allows computing device 200 to communicate with othercomputing devices via a network, e.g., a local area network (LAN), awide area network (WAN) or any suitable combination thereof). Networkinterface 206 thus includes any necessary hardware for communicatingover such networks, such as radios, network interface controllers (NICs)and the like. Computing device 200 also includes an input/outputinterface 208, including the necessary hardware for interconnectingprocessor 202 with various input and output devices. Interface 208 caninclude, among other components, a universal serial bus (USB) port, anaudio port for sending and receiving audio data, a video graphics array(VGA), a digital visual interface (DVI) or other port for sending andreceiving display data, and any other suitable components.

Still referring to FIG. 2, via interface 208, computing device 200 isconnected to input devices including a keyboard and mouse 210, amicrophone 212, as well as scope 116 and tracking system 112, mentionedabove. Similarly, computing device 200 can be connected to theadditional imaging devices mentioned above via interface 208. Also viainterface 208, computing device 200 is connected to output devicesincluding illumination or projection components 214, e.g., lights,projectors and the like, as well as display 110 and robotic arm 114mentioned above. Other input, e.g., touch screens, and output devices,e.g., speakers, are also encompassed. The I/O interface 208 may beomitted entirely in some embodiments, or may be used to connect to onlya subset of the devices mentioned above. The remaining devices may beconnected to computing device 200 via network interface 206.

Still referring to FIG. 2, the computing device 200 stores, in memory204, an imaging device tracking application 216 (also referred to hereinas application 216) comprising a plurality of computer readableinstructions executable by processor 202. When processor 202 executesthe instructions of application 216 (or, indeed, any other applicationstored in memory 204), processor 202 performs various functionsimplemented by those instructions, as will be discussed below. Processor202, or computing device 200 more generally, is therefore said to be“configured” or “operating” to perform those functions via the executionof application 216.

Still referring to FIG. 2, also stored in memory 204 are various datarepositories, including a patient data repository 218. Patient datarepository 218 can contain a surgical plan defining the various steps ofthe minimally invasive surgical procedure to be conducted on patient104, as well as image data relating to patient 104, such as MRI and CTscans, three-dimensional models of the brain of patient 104, and thelike.

Still referring to FIG. 2, as mentioned above, computing device 200 isconfigured, via the execution of application 216 by processor 202, toperform various functions to capture intraoperative images depictingcertain areas of patient 104 with each other and with preoperativeimages, in order to track the position of the above-mentioned additionalimaging devices. Those functions will be described in further detailbelow.

Referring to FIG. 3, this diagram illustrates a method 300 of trackingsurgical imaging devices, in accordance with an embodiment of thepresent disclosure. The method 300 is below described in connection withits performance, particularly by computing device 200, in the operatingtheatre 100. In other words, the computing device 200 is configured toperform the steps of the method 300 via the execution of theinstructions of the application 216 by the processor 202. At step 305,computing device 200 is configured to store a preoperative image ofpatient tissue registered to a frame of reference of a tracking systemconnected to the computing device. In the present example, as notedabove, the patient tissue in question is the brain of patient 104. Inother embodiments, the patient tissue can be any of a wide variety ofother organs, tissues, portions thereof or combinations thereof. In thepresent example, the preoperative image stored at step 305 is an MRIimage. The MRI image can be captured in any suitable manner, e.g., byoperating an MRI scanner within or outside operating theatre 100, andprovided to computing device 200 for storage at step 305 in memory 204,e.g., in the repository 218.

Referring to FIG. 4, this diagram illustrates a preoperative image 400,in accordance with an embodiment of the present disclosure. In thepresent example, the image 400 depicts the entire brain of patient 104,as well as certain surrounding anatomical structures (such as the skulland eyes). This is not necessary—as noted previously, the preoperativeimage stored at step 305 can depict any of a wide variety of patienttissues (FIG. 3). In addition, the preoperative image 400 need not be anMRI scan. Any suitable imaging modality, e.g., computed tomography (CT),ultrasound, photo-acoustic tomography, positron emission tomography(PET) imaging and the like, can be employed to capture the image storedat step 305.

Still referring to FIG. 4, as mentioned above, the preoperative image,stored at step 305, is registered to a frame of reference. In thepresent example, the preoperative image 400 is registered to frame ofreference 113 of tracking system 112. In other words, each pixel orvoxel in preoperative image 400 is associated with a physical positionwithin operating theatre 100, defined in relation to the point of originof frame of reference 113. For example, the computing device 200 storespositional data within the image file for image 400, indicating thecorresponding position within frame of reference 113 for each pixel orvoxel. In other embodiments, computing device 200 stores, within theimage file for the image 400, an indication of the starting coordinateswithin the frame of reference 113 for the first pixel or voxel of theimage 400 and an indication of the spacing of the pixels or voxelswithin the frame of reference 113, e.g., the first voxel is positionedat the coordinates 130.1 cm, 25.4 cm, 81.3 cm in frame of reference 113,wherein each voxel represents a distance of 0.5 mm in each axis of frameof reference 113.

Still referring to FIG. 4, the process by which preoperative image 400is registered to the frame of reference 113 is not limited. For example,in some embodiments, an operator, e.g., the surgeon 102, holds a trackedtool over a specific point in the preoperative image 400 thatcorresponds to a specific location on the patient 104 that bears amarker. In other words, the tracking system 112 is manually instructedregarding the corresponding points between the preoperative image 400and the patient 104.

Referring back to FIG. 3, proceeding to step 310, during the medicalprocedure (that is, intraoperatively), computing device 200 isconfigured to receive, from a first imaging device, a firstintraoperative image of a first region of the patient tissue. The imagecan be received by way of a control signal from the computing device200, thereby causing the imaging device to capture the image. In otherembodiments, the image is received at the computing device 200 followinga command from an operator, e.g., the surgeon 102, to the imaging deviceto capture the image.

Referring tack to FIG. 4, the first intraoperative image, in general,depicts a region of the patient tissue that is smaller than the regiondepicted by the preoperative image. In addition, the firstintraoperative image has a greater pixel density than the preoperativeimage (or voxel density—the term “pixel density” is used herein toindicate density of either pixels or voxels, depending on whether therelevant images are two-dimensional or three-dimensional). That is, thefirst intraoperative image represents a given physical area with alarger number of pixels or voxels than preoperative image 400, and,thus, depicts the patient tissue in the first region in greater detailthan preoperative image 400.

Referring to FIG. 5A, this diagram illustrates a first intraoperativeimage 500, in accordance with an embodiment of the present disclosure.In particular, intraoperative image 500 is an image captured by scope116 mounted on robotic arm 114 and provided to computing device 200. Atstep 315, the computing device 200 is configured to receive a positionof the first imaging device, e.g., scope 116, from tracking system 112,and based on the position, to register intraoperative image 500 with theframe of reference 113. Registering intraoperative image 500 with frameof reference 113 includes storing coordinates within frame of reference113 in image 500, or (as mentioned above in connection with theregistration of preoperative image 400) storing an indication of thelocation of the first pixel of the image 500 in the frame of reference113, and an indication of the distance in the frame of reference 113between each pixel. The registration at step 315 can be performedautomatically by computing device 200. For example, given that theposition of scope 116 is known from the tracking system 112 and that theimaging parameters (field of view, focal length and the like) are alsoknown, the computing device 200 determines the position of the image 500in the operating theatre 100.

Referring to FIG. 5B, this diagram illustrates results of registrationis provided, in accordance with an embodiment of the present disclosure.As noted earlier, the preoperative image 400 is registered to frame ofreference 113. Thus, when the intraoperative image 500 is registered tothe frame of reference 113, the intraoperative image 500 can be overlaidon the preoperative image 400 to indicate what region of patient tissueis depicted by both images. In particular, a region 504 identifies thetissue that is depicted by both the images 400 and 500. At step 320, thecomputing device 200 is configured to determine whether any furtherimaging devices are active in the operating theatre 100 (FIG. 3). Forexample, the processor 202 is configured to identify any such imagingdevices that are connected to the computing device 200 via the interface208. Step 320 further comprises determining whether such devices areenabled (as opposed to being connected but currently disabled).

Still referring to FIG. 5B and referring back to FIG. 3, when thedetermination at step 320 is affirmative, the performance of the method300 proceeds to step 325. At step 325, the computing device 200 isconfigured to receive a subsequent intraoperative image from asubsequent imaging device connected to the computing device 200. Thedetermination, at step 320, can be combined with step 325, in that, whenan intraoperative image is received from a subsequent imaging device,the determination at step 320 is affirmative.

Still referring to FIG. 5B and referring back to FIG. 3, the subsequentintraoperative image (which may also be referred to as the secondintraoperative image in the present example performance of the method300), received at step 325, depicts a subsequent region of the patienttissue, e.g., of the brain of patient 104. In general, the subsequentregion is smaller than the previous region; and, thus, in the presentperformance of step 325, the region depicted in the subsequentintraoperative image is smaller than the region depicted inintraoperative image 500. In addition, the subsequent image has agreater pixel density than the previous image.

Referring to FIG. 6A, this diagram illustrates a second intraoperativeimage 600, in accordance with an embodiment of the present disclosure.In the present example, second intraoperative image 600 is captured byusing an OCT probe, although, in other embodiments, any of a widevariety of imaging devices can be employed to capture the secondintraoperative image. For example, in some embodiments the secondintraoperative image may be captured with another optical surgicalscope, similar to scope 116 with the exception that the second scope isconfigured to capture higher-resolution images of smaller regions ofpatient tissue than scope 116.

Referring back to FIG. 3, at step 330, computing device 200 isconfigured to register subsequent intraoperative image 600 to theprevious intraoperative image (image 500, in this case). Theregistration of image 600 to image 500 can be conducted by computingdevice 200 in a variety of ways, For example, computing device 200employs conventional feature-based image registration techniques,intensity-based image registration techniques, or a combination thereof.In other embodiments, the above mentioned techniques can be supplementedor replaced by registration techniques, such as those described inApplicant's co-pending PCT Application No. PCT/CA2014/000849, filed Nov.27, 2014, and entitled “Method, System and Apparatus for QuantitativeSurgical Image Registration,” which is herein incorporated by reference.In some embodiments, further image registration techniques may beapplied at step 330, such as those described in Applicant's co-pendingPCT Application No. PCT/CA2015/000011, filed Jan. 7, 2015, and entitled“Method, System and Apparatus for Adaptive Image Acquisition,” which isherein incorporated by reference.

Still referring back to FIG. 3, in further embodiments, additional imageregistration techniques can be employed, in addition to or instead ofthose mentioned above. For example, images, captured with OCT or Ramanspectroscopy probes, are registered to images captured with a surgicalscope by detecting, within the surgical scope images, a visible pointeremitted by the OCT or Raman spectroscopy probes. The pointer, e.g., aphysical pointer or a beam of light, has a known location within theprobe images (based on the physical geometry of the probe itself). Thus,when the pointer is detected in the scope images, e.g., a location forthe pointer is established within the scope images, the probe images andscope images are registered. More generally, each successive toolgenerates light or other output that is detectable to the previous tool(even if the output is not visible to operators or to scope 116).

Still referring back to FIG. 3, as another example, a probe (such as anOCT probe) may be placed in direct contact with patient tissue, theprobe may cause deformation of the tissue in the form of a shallowaround the probe. Computing device 200 can be configured to detect sucha shallow, and register the images based on the location of the shallow(and, by implication, the location of the probe) within the scopeimages.

Still referring back to FIG. 3, following the completion of step 330,computing device 200 is configured to store the registered image 600 inmemory 304 (for example, in repository 218). The image can be storedwith registration data generated during the performance of step 330. Forexample, each pixel or voxel of image 600 comprises positional dataidentifying either a corresponding location in in 500 or a positionwithin frame of reference 113. In other examples, image 600 comprisesdata indicating the location within image 500 or frame of reference 113of the first pixel of image 600, as well as the distance in image 500 orin frame of reference 113 between the pixels of image 600.

Referring to FIG. 6B, this diagram illustrates an intraoperative image500 with a subsequent region 604 of patient tissue illustrated thereon,in accordance with an embodiment of the present disclosure. The locationand size of region 604 is determined by the registration process at step330 (FIG. 3). As mentioned above, the image registration process at step330 can yield position information for the image 600 based on frame ofreference 113, despite the subsequent imaging device not being trackedby the tracking system 112 (unlike scope 116). To determine positionaldata for image 600, computing device 200 is configured to register theimage 600 to the image 500, e.g., based on image features or othertechniques as mentioned above, to derive the position of the image 600within the image 500, e.g., in terms of pixel-based coordinates. Sincethe image 500 is already registered to the frame of reference 113, theregistration of the image 600 to the image 500 is then be converted to aregistration of the image 600 to the frame of reference 113.

Referring to FIG. 7A, this diagram illustrates a third intraoperativeimage received in the method, as shown in FIG. 3, in accordance with anembodiment of the present disclosure. Having completed the performanceof step 330, computing device 200 returns to step 320 to determinewhether any further imaging devices are active. In the present example,assumed is that the determination at step 320 is again affirmative; andthrough a further performance of steps 325 and 330, a thirdintraoperative image 700 is received at computing device from a thirdimaging device such as a Raman spectroscopy probe.

Referring to FIG. 7B, this diagram illustrates a registration of thethird intraoperative image 700 to the second intraoperative image 600,as shown in FIG. 7A, in accordance with an embodiment of the presentdisclosure. The third intraoperative image 700 is registered to theprevious intraoperative image (image 600), and is, thus, determined torepresent a region 704 of patient tissue that is smaller than the regiondepicted by image 600. As shown in FIGS. 7A and 7B, the thirdintraoperative image 700 has a greater pixel density than the secondintraoperative image 600.

Referring to FIGS. 7A and 7B, together, and referring back to FIG. 3,following the second performance of steps 325 and 330, computing device200, again, returns to step 320. In the present example, assumed is thatin this third performance of step 320, the determination is negative(that is, there are no further imaging devices active). In otherembodiments, the performance of steps 325 and 330 can be repeated anynumber of times, depending on the imaging devices in use in operatingtheatre 100. Although each intraoperative image in the examplesdescribed above was captured using a different imaging modality, this isnot necessary. In other performances of method 300, imaging devicesusing the same imaging modality (albeit it increasing pixel densities)can be employed to capture some or all of the intraoperative images.

Referring to FIGS. 7A and 7B, together, and referring back to FIG. 3,responsive to a negative determination at step 320, computing device 200is configured to proceed to step 335. At step 335, computing device 200is configured to control display 110 to present preoperative image 400,overlaid with the intraoperative images in the sequence in which theywere captured. The sets of images, presented at step 335, are presentedin real-time, or near real-time, as intraoperative images are receivedat computing device. In addition, step 335 can be performed at a latertime (postoperatively, for example) by retrieving the stored registeredimages and presenting them on display 110 (or indeed, on any othersuitable display outside operating theatre 100).

Referring to FIG. 8, this diagram illustrates performance of the step335 of displaying images overlaid in sequence of capture in the method300, as shown in FIG. 3, in accordance with an embodiment of the presentdisclosure. Various forms of overlaying can be employed at step 335(FIG. 3). For example, the images can be presented at different scales),with indications such as lead lines indicating which region of one imageis depicted by the next image, In other examples, all images can beshown at the same scale and the images can be overlaid directly on eachother. In still other examples, the latest image (that s, the imagehaving the highest pixel density) can be shown on display 110, and onlyportions of the previous images can be shown beneath the latest image.

Still referring to FIG. 8 and referring back to FIG. 3, furtheroperations can also be performed at step 335, or after step 335, bycomputing device 200. For example, computing device 200 can receive aselection of a portion of any of the images described above and shown ondisplay 110. In response to such a selection computing device 200 can beconfigured to transmit a command to a robotic arm supporting a furtherimaging device, a tool, such as a laser emitter or the like. The commandcan be a command to position the instrument supported by the robotic armat the location in frame of reference 113 corresponding to the selectedportion of the image. Computing device 200 determines the location inframe of reference 113 by, as above mentioned, employing imageregistration information determined at step 330 as well as theregistration of the first intraoperative image with frame of reference113.

Still referring to FIG. 8, position data for communication to a roboticdevice may be derived with greater accuracy from image registrationdata, e.g., between images 500 and 600) than can be provided by trackingsystem 112. For certain surgical instruments (such as laser ablationtools, for example), such increased accuracy may be required tocorrectly position the instruments. A robotic positioning device, incombination with the performance of method 300, may provide suchaccuracy where the tracking system 112 cannot. If the surgical tool oreffector is at least partly controlled by a robotic arm, such increasedaccuracy also enables the robotic arm to limit the possible affectingarea of the surgical tool. This can prevent certain important areas ofthe organ from being damaged unintentionally activities during theprocedure.

Still referring to FIG. 8, a wide variety of other positioning devicesand tools are contemplated. For example, devices such as microscopic ornear-microscopic robots (also referred to as nanobots) may be employedto perform various tasks during the surgical procedure. The performanceof method 300 can provide more accurate positioning information for suchrobots (arid any associated positioning systems used to deploy them)than tracking system 112. Examples of high-accuracy robotic positioningsystems include electromagnetically-driven systems such as thatdescribed by Kummer et al., in “OctoMag: An Electromagnetic System for5-DOF Wireless Micromanipulation,” IEEE Transactions on Robotics, 26(6)2010. Further examples include parallel kinematic positioning devices,also referred to as hexapods (see http://www.emdt.co.uk/article/hexapodsand http://biotsavart.tripod.com/hexapod.htm).

Still referring to FIG. 8 and referring back to FIG. 3, the computingdevice 200 can also perform the above-mentioned instruction of roboticsupporting devices at step 335. For example, a selection of a locationon image 500 can be received at processor 202, e.g., from input devicessuch as keyboard and mouse 210; and, in response, the computing device200 can direct a robotic arm, supporting the Raman spectroscopic probe,to the selected location. Adjustments to the location of the probebefore the completion of step 325, e.g., before the capture of an imagewith the probe, can be performed via the receipt of control inputs atthe computing device 200 from an operator such as surgeon 102.

Referring to FIG. 9, this diagram illustrates a sequence of imagingmodalities that can be linked via the performance of method 300 during asurgical procedure, in accordance with an embodiment of the presentdisclosure. Successively more detailed imaging modalities, e.g., havingfiner resolutions, may be employed, with each modality being registeredto, e.g., tracked by, the images generated by the previous modality viathe performance of method 300. Examples of the level of biologicaldetail with which each modality is generally concerned as well asexample resolutions for each modality are shown.

Referring to FIG. 10, this diagram illustrates relationships betweenpositioning devices, e.g., robotic arm 114, visualization devices, e.g.,scope 116, and effectors, e.g., resectors and the like, in accordancewith an embodiment of the present disclosure. Positioning devices canposition both effectors (dashed lines) and visualization devices(dot-dashed lines), while visualization devices can provide tracking ofeffectors (solid lines) to provide feedback to the positioning devices,via the performance of method 300 by computing device 200.

Referring back to FIGS. 1-10, variations to the above-mentioned systemsand methods are contemplated. For example, although individual imagesare described and shown in the drawings, method 300 can also be appliedto video streams received at computing device 200 from imaging devices.In some embodiments, each image, e.g., either preoperative,intraoperative or postoperative images, captured by the system aretagged with unique identifiers, such as metadata tags, in the imageheader. These images are also cross-referenced and/or linked tosubsequent images at various biological levels, as shown in FIG. 9.These images may be stored on a local imaging database in memory 204 orremotely from computing device 200, for example, on a networked picturearchiving and communication system (PACS) or digital imaging andcommunications in medicine (DICOM) server. These images and databasesmay also be connected to an informatics platform where these tagged andindexed images can be used for subsequent clinical studies orprocedures. Further, analytical analysis can be conducted on thedatabases or informatics systems to identify trends or commonalitiesamong the different images.

Tagged and indexed images also enables the ability to match the level ofaccuracy between the various visualization system, imaging modalities,effector and device positioning system to maximize the effectiveness ofall the tools used. Users/operators may also be able to providefeedback, comment or rate images based on quality metrics. For example,a user may rate an OCT image, using a robotic arm and a resector toprovide “great wide-field view, mediocre magnification,” or second imageusing Raman, a micron precision system and a nano-robot to provide“limited view, but excellent magnification.”

The scope of the claims should not be limited by the embodiments setforth in the above examples, but should be given the broadestinterpretation consistent with the present disclosure as a whole.

What is claimed:
 1. A method of tracking positions of surgical imagingdevices by way of a computing device comprising a processor, the methodcomprising: by using the processor, receiving a first intraoperativeimage of a first area of a patient from a first imaging device, thefirst intraoperative image having a first intraoperative imageresolution finer than a preoperative image resolution of a preoperativeimage; receiving a position of the first imaging device in a coordinatesystem from an optical tracking system; receiving a secondintraoperative image of a second area of the patient from a secondimaging device, the second intraoperative image depicting the secondarea within the first area, and the second intraoperative image having asecond intraoperative image resolution finer than first intraoperativeimage resolution; registering a second intraoperative image with thefirst intraoperative image, thereby providing a registration of thesecond intraoperative image with the first intraoperative image;determining a position of a second imaging device in the coordinatesystem based on the registration of the second intraoperative image withthe first intraoperative image; determining whether a subsequent imagingdevice is detected, and when the subsequent imaging device is detected,receiving a subsequent intraoperative image of a subsequent area of thepatient from the subsequent imaging device, the subsequentintraoperative image depicting the subsequent area within a previousarea, and the subsequent intraoperative image having a subsequentintraoperative image resolution finer than that of a previousintraoperative image resolution; registering the subsequentintraoperative image with a previous intraoperative image, therebyproviding a registration of the subsequent intraoperative image with theprevious intraoperative image; and determining a position of thesubsequent imaging device in the coordinate system based on theregistration of the subsequent intraoperative image with the previousintraoperative image; and otherwise, controlling a display to presentthe preoperative image overlaid with the first intraoperative image, thefirst intraoperative image overlaid with second intraoperative image,based on the position of the second imaging device, the previousintraoperative image overlaid with the subsequent intraoperative image,and each of the first intraoperative image and the second intraoperativeimage related to a distinct imaging modality.
 2. The method of claim 1,further comprising: obtaining the preoperative image of the patient, thepreoperative image having the preoperative image resolution; registeringthe preoperative image to the coordinate system corresponding to anoperating facility; establishing a connection between the computingdevice and: the first imaging device having a first resolution, a firstfield of view, and a plurality of reflective markers; the opticaltracking system configured to track the first imaging device in thecoordinate system by detecting the plurality of reflective markers, thefield of view of the optical tracking system greater than the field ofview of the first imaging device; and the second imaging device having asecond field of view, the second field of view less than the first fieldof view; establishing a connection between the computing device and athird imaging device having a third field of view, the third field ofview less than the second field of view; receiving a thirdintraoperative image of a third area of the patient from the thirdimaging device, the third intraoperative image depicting the third areawithin the second area, the third intraoperative image having a thirdintraoperative image resolution finer than the second intraoperativeimage resolution, and the third intraoperative image related to animaging modality distinct from the first intraoperative image and thesecond intraoperative image; registering the third intraoperative imagewith the second intraoperative image, thereby providing a registrationof the third intraoperative image with the second intraoperative image;and determining a position of the third imaging device in the coordinatesystem based on the registration of the third intraoperative image withthe second intraoperative image, the registration of the secondintraoperative image with the first intraoperative image, and theposition of the first imaging device.
 3. The method of claim 1, furthercomprising registering the first intraoperative image with thepreoperative image by identifying corresponding image features in thefirst intraoperative image and the preoperative image.
 4. The method ofclaim 2, wherein receiving the first intraoperative image, receiving thesecond intraoperative image, and receiving the third intraoperativeimage respectively comprise each of the first imaging device, the secondimaging device, and the third imaging device implementing a distinctimaging modality.
 5. The method of claim 2, wherein obtaining thepreoperative image comprises obtaining an MRI image.
 6. The method ofclaim 1, further comprising: receiving a selection of a location withinthe second intraoperative image; and determining a position of theselected location within the coordinate system based on the registrationof the second intraoperative image with the first intraoperative image.7. The method of claim 2, wherein registering the third intraoperativeimage with the second intraoperative image is based on detection of apointer by the third imaging device in the second intraoperative image.8. The method of claim 2, wherein establishing the connection betweenthe computing device and the first imaging device the comprisesestablishing a connection between the computing device and an opticalscope.
 9. The method of claim 2, wherein establishing the connectionbetween the computing device and the second imaging device comprisesestablishing a connection between the computing device and an opticalcoherence tomography (OCT) probe, and wherein establishing theconnection between the computing device and the third imaging devicecomprises establishing a connection between the computing device and aspectroscopic probe.
 10. The method of claim 2, further comprisingestablishing a connection between the computing device and a remoteimaging database, the remote imaging database operable with at least oneof a networked picture archiving and communication system (PACS) and adigital imaging and communications in medicine (DICOM) server.
 11. Acomputing device comprising a processor, the processor configured to:receive a first intraoperative image of a first area of a patient from afirst imaging device, the first intraoperative image having a firstintraoperative image resolution finer than a preoperative imageresolution of a preoperative image; receive a position of the firstimaging device in a coordinate system from an optical tracking system;receive a second intraoperative image of a second area of the patientfrom a second imaging device, the second intraoperative image depictingthe second area within the first area, and the second intraoperativeimage having a second intraoperative image resolution finer than that ofthe first intraoperative image resolution; register the secondintraoperative image with the first intraoperative image, therebyproviding a registration of the second intraoperative image with thefirst intraoperative image; determining a position of the second imagingdevice in the coordinate system based on the registration of the secondintraoperative image with the first intraoperative image; determinewhether a subsequent imaging device is detected, and when the subsequentimaging device is detected, receive a subsequent intraoperative image ofa subsequent area of the patient from the subsequent imaging device, thesubsequent intraoperative image depicting the subsequent area within aprevious area, and the subsequent intraoperative image having asubsequent intraoperative image resolution finer than a previousintraoperative image resolution; register the subsequent intraoperativeimage with a previous intraoperative image, thereby providing aregistration of the subsequent intraoperative image with the previousintraoperative image; and determine a position of the subsequent imagingdevice in the coordinate system based on the registration of thesubsequent intraoperative image with the previous intraoperative image;and otherwise, control a display to present the preoperative imageoverlaid with the first intraoperative image, the first intraoperativeimage overlaid with second intraoperative image, based on the positionof the second imaging device, the previous intraoperative image overlaidwith the subsequent intraoperative image, and each of the firstintraoperative image and the second intraoperative image related to adistinct imaging modality.
 12. The computing device of claim 11, whereinthe processor is further configured to: obtain the preoperative image ofthe patient, the preoperative image having the resolution; register thepreoperative image to the coordinate system corresponding to anoperating facility; establish a connection between the computing deviceand: the first imaging device having a first resolution, a first fieldof view, and a plurality of reflective markers; the optical trackingsystem configured to track the first imaging device in the coordinatesystem by detecting the plurality of reflective markers, the field ofview of the optical tracking system greater than the field of view ofthe first imaging device; and the second imaging device having a secondfield of view, the second field of view less than the first field ofview; establish a connection between the computing device and a thirdimaging device having a third field of view, the third field of viewless than the second field of view; receive a third intraoperative imageof a third area of the patient from the third imaging device, the thirdintraoperative image depicting the third area within the second area,the third intraoperative image having a third intraoperative imageresolution finer than the second intraoperative image resolution, andthe third intraoperative image related to an imaging modality distinctfrom the first intraoperative image and the second intraoperative image;register the third intraoperative image with the second intraoperativeimage, thereby providing a registration of the third intraoperativeimage with the second intraoperative image; and determine a position ofthe third imaging device in the coordinate system based on theregistration of the third intraoperative image with the secondintraoperative image, the registration of the second intraoperativeimage with the first intraoperative image, and the position of the firstimaging device.
 13. The device of claim 11, wherein the processor isfurther configured to register the first intraoperative image with thepreoperative image by identifying corresponding image features in thefirst intraoperative image and the preoperative image.
 14. The device ofclaim 12, wherein the processor is configured to respectively receivethe first intraoperative image, receive the second intraoperative image,and receive the third intraoperative image from each of the firstimaging device, the second imaging device, and the third imaging deviceimplementing a distinct imaging modality.
 15. The device of claim 12,wherein the processor is configured to obtain the preoperative image byobtaining an MRI image.
 16. The device of claim 11, wherein theprocessor is further configured to: receive a selection of a locationwithin the second intraoperative image; and determine a position of theselected location within the coordinate system based on the registrationof the second intraoperative image with the first intraoperative image.17. The device of claim 12, wherein the processor is configured toregister the third intraoperative image with the second intraoperativeimage based on detection of a pointer by the third imaging device in thesecond intraoperative image.
 18. The device of claim 12, wherein theprocessor is configured to establish the connection between thecomputing device and the first imaging device by establishing aconnection between the computing device and an optical scope.
 19. Thedevice of claim 12, wherein the processor is configured to establish theconnection between the computing device and the second imaging device byestablishing a connection between the computing device and an opticalcoherence tomography (OCT) probe, wherein the processor is configured toestablish the connection between the computing device and the thirdimaging device by establishing a connection between the computing deviceand a spectroscopic probe, and wherein the processor is configured toestablish a connection between the computing device and a remote imagingdatabase, the remote imaging database operable with at least one of anetworked picture archiving and communication system (PACS) and adigital imaging and communications in medicine (DICOM) server.
 20. Amethod of providing a computing device for tracking positions ofsurgical imaging devices, the method comprising providing a processorconfigured to: receive a first intraoperative image of a first area of apatient from a first imaging device, the first intraoperative imagehaving a first intraoperative image resolution finer than that of apreoperative image; receive a position of the first imaging device in acoordinate system from an optical tracking system; receive a secondintraoperative image of a second area of the patient from a secondimaging device, the second intraoperative image depicting the secondarea within the first area, and the second intraoperative image having asecond intraoperative image resolution finer than that of the firstintraoperative image resolution; register the second intraoperativeimage with the first intraoperative image, thereby providing aregistration of the second intraoperative image with the firstintraoperative image; determining a position of the second imagingdevice in the coordinate system based on the registration of the secondintraoperative image with the first intraoperative image; determinewhether a subsequent imaging device is detected, and when the subsequentimaging device is detected, receive a subsequent intraoperative image ofa subsequent area of the patient from the subsequent imaging device, thesubsequent intraoperative image depicting the subsequent area within aprevious area, and the subsequent intraoperative image having asubsequent intraoperative image resolution finer than a previousintraoperative image resolution; register the subsequent intraoperativeimage with a previous intraoperative image, thereby providing aregistration of the subsequent intraoperative image with the previousintraoperative image; and determine a position of the subsequent imagingdevice in the coordinate system based on the registration of thesubsequent intraoperative image with the previous intraoperative image;and otherwise, control a display to present the preoperative imageoverlaid with the first intraoperative image, the first intraoperativeimage overlaid with second intraoperative image, based on the positionof the second imaging device, the previous intraoperative image overlaidwith the subsequent intraoperative image, and each of the firstintraoperative image and the second intraoperative image related to adistinct imaging modality.